What Your Peers Struggle The Most?¶
Find out what topics and discussion materials is common among your peers.
Does "Operating Systems", "ubuntu", "memory", "page" sound familiar? Looks like you are not the only one.
Explore how we reach to that conclusion
It's you that telling us how to do it. We collect your responses, add some magic salt (NLP), and kaboom. Here is the detailed path:
- We fetch the topics and discussion materials in a text format.
- We use NLP for extracting the stem of each word.
- We count how frequently the tags appear.
- We implement back a spellchecker (Don't forget, they were just stems).
- Combine the results, and visualize.
Stemming Words¶
In [146]:
from nltk.stem import PorterStemmer
from nltk.tokenize import sent_tokenize, word_tokenize
import pickle
import collections
from spellchecker import SpellChecker
spell = SpellChecker()
ps = PorterStemmer()
In [147]:
with open('topics.pk', 'rb') as handle:
topics_list = pickle.load(handle)
In [149]:
counts = {}
doc.keys()
for doc in topics_list:
print(doc["topic_name"])
words = word_tokenize(doc["topic_name"])
for word in words:
word = ps.stem(word.lower())
word = spell.correction(word) if not len(spell.candidates(word))>1 else word
if word in counts:
counts[word] += 1
else:
counts[word] = 1
# print(doc["discussion_material"])
sorted_counts = sorted(counts.items(), key=lambda x: x[1], reverse=True)
In [151]:
sorted_counts
Out[151]:
In [152]:
stop_words = ['?', 'explain', 'what', 'is', 'the', 'a', 'of',
'in', 'are', 'an', '.', 'it', 'for', 'thi', 'with',
'be', '–', 'how', 'to', 'when', 'and', 'you', 'can',
',', 'from', 'better', 'do', 'that', 'by', 'on', 'as',
'or', ')', '('
]
In [153]:
for i in stop_words:
counts[i]=0
sorted_counts = sorted(counts.items(), key=lambda x: x[1], reverse=True)
In [154]:
sorted_counts
Out[154]:
Discussion Material¶
In [155]:
counts_disc = {}
for doc in topics_list:
print(doc["discussion_material"])
words = word_tokenize(doc["discussion_material"])
for word in words:
word = ps.stem(word.lower())
word = spell.correction(word) if not len(spell.candidates(word))>1 else word
if word in counts_disc:
counts_disc[word] += 1
else:
counts_disc[word] = 1
In [156]:
for i in stop_words:
counts_disc[i]=0
sorted_counts_disc = sorted(counts_disc.items(), key=lambda x: x[1], reverse=True)
In [157]:
sorted_counts_disc
Out[157]:
In [ ]:
WORD CLOUD¶
In [111]:
# Start with loading all necessary libraries
import numpy as np
import pandas as pd
from os import path
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
import matplotlib.pyplot as plt
In [112]:
?WordCloud
In [158]:
create_para = []
for i in sorted_counts:
create_para.extend( [i[0]] * i[1])
# create_para = create_para + (" " + i[0]) * i[1]
create_para
Out[158]:
In [167]:
# # Start with one review:
# text = df.description[0]
# Create and generate a word cloud image:
wordcloud = WordCloud(collocations=False).generate(" ".join(create_para))
# Display the generated image:
plt.figure(figsize=(100,50))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.savefig("worldcloud_topics.png")
plt.show()
In [163]:
create_para_disc = []
for i in sorted_counts_disc:
create_para_disc.extend( [i[0]] * i[1])
create_para_disc
Out[163]:
In [166]:
# Create and generate a word cloud image:
wordcloud_disc = WordCloud(collocations=False).generate(" ".join(create_para_disc))
# Display the generated image:
plt.figure(figsize=(100,50))
plt.imshow(wordcloud_disc, interpolation='bilinear')
plt.axis("off")
plt.savefig("worldcloud_materials.png")
plt.show()
Word Enhancement - Spell Checking¶
Some of the words like "Operating System" is stemmed as "oper" "system". There has to be a novel way to catch these. This is why we try to look at pyspellchecker (Looks like hunspell is not supported anymore)
In [117]:
from spellchecker import SpellChecker
spell = SpellChecker()
# find those words that may be misspelled
misspelled = spell.unknown(['something', 'is', 'hapenning', 'here'])
for word in misspelled:
# Get the one `most likely` answer
print(spell.correction(word))
# Get a list of `likely` options
print(spell.candidates(word))
In [145]:
spell.correction("provid")
spell.candidates("ubuntu")
Out[145]:
In [ ]:
# Updated above (whole) code with spellchecker.